Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus includes a change unit configured to change a driving state of an image sensor for capturing an image signal in response to an operation for changing the driving state of the image sensor, and a flicker correction unit configured to correct an image signal based on a flicker component detected from image signals of a frame captured a predetermined number of frames before the frame of the image signal to be corrected, wherein the flicker correction unit is configured to change the predetermined number when the driving state of the image sensor is changed by the change unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique to correct a flickercomponent in an image captured using an image sensor.

2. Description of the Related Art

In an imaging apparatus employing an image sensor, periodic light anddark changes may appear in a captured image when an object underfluorescent lighting is continuously photographed for a plurality offrames (or fields) as a moving image or continuous still images. Thisphenomenon is referred to as a fluorescent flicker, which occurs due toa difference between the frequency of the commercial power supply and avertical synchronizing frequency of the imaging apparatus.

Conventionally, a technique has been proposed which detects a periodicfluctuation component of a signal level (hereafter referred to as aflicker component) and corrects a flicker by applying the inverse gainof the detected flicker component to the image signal (refer to JapanesePatent Application Laid-Open No. 11-122513).

The flicker component of a fluorescent flicker in an imaging apparatusvaries depending on the frequency of the power supply and the lightemission properties. Furthermore, a driving state of an imagingapparatus, such as accumulation time and a frame rate of an image sensorand a change in reading time due to an adding process and a thinningprocess of pixels, also causes variation in the flicker component. Thedriving state of an imaging apparatus is changed or controlled by animaging system or in response to instructions from a user. Therefore,the driving state can rapidly change. In this case, the flickercomponent can significantly change with a change in the driving state.

However, a correction method discussed in Japanese Patent ApplicationLaid-Open No. 11-122513 is a method of detecting a flicker componentusing an intensity distribution in the vertical direction generated byintegrating an image signal of the current and past frames (or fields)in the horizontal direction. In such a method, it is difficult to followan abrupt change in flicker component and to appropriately correct aflicker, because a time difference occurs between an image signal usedto detect the flicker and an image signal to be corrected. Thus, it isdifficult to execute suitable correction for a flicker. On the contrary,the correction can sometimes adversely affect a captured image.

SUMMARY OF THE INVENTION

Exemplary embodiments of the present invention are directed to an imageforming apparatus and an image processing method capable of correcting aflicker component precisely even if the flicker component is abruptlychanged due to a change in driving method of an image sensor.

According to an aspect of the present invention, an image processingapparatus includes a change unit configured to change a driving state ofan image sensor for capturing an image signal in response to anoperation for changing the driving state of the image sensor, and aflicker correction unit configured to correct an image signal based on aflicker component detected from image signals of a frame captured apredetermined number of frames before the frame of the image signal tobe corrected, wherein the flicker correction unit is configured tochange the predetermined number when the driving state of the imagesensor is changed by the change unit.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram illustrating a configuration of a flickerdetection and correction apparatus according to a first exemplaryembodiment of the present invention.

FIG. 2 is a timing chart illustrating the timing from flicker detectionto flicker correction according to the first exemplary embodiment.

FIGS. 3A and 3B are schematic views illustrating a state of a flicker inan image signal due to an accumulation time difference according to thefirst exemplary embodiment.

FIG. 4 is a schematic view illustrating a relationship betweenaccumulation time and modulation of a luminance signal caused by aflashing light source according to the first exemplary embodiment.

FIG. 5 is a timing chart illustrating a relationship betweenaccumulation time and phase difference according to the first exemplaryembodiment.

FIG. 6 is a timing chart illustrating the timing from flicker detectionto flicker correction according to a second exemplary embodiment of thepresent invention.

FIG. 7 is a timing chart illustrating the timing from flicker detectionto flicker correction according to a third exemplary embodiment of thepresent invention.

FIGS. 8A and 8B are schematic views illustrating the state of a flickerin an image signal due to a reading time difference according to thethird exemplary embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 is a block diagram illustrating a flicker detection andcorrection apparatus serving as an image processing apparatus accordingto a first exemplary embodiment of the present invention.

An image sensor 101 includes a photoelectric conversion elements. Theimage sensor 101 is driven by a driving pulse generated based on acontrol signal that is generated by a control unit 102, and converts anobject image into an electric signal by photoelectric converting, andoutputs the converted analog signal to an AD conversion unit 106. The ADconversion unit 106 converts the input analog signal into a digitalsignal and outputs it as a digital signal. At this time, if the imagesensor 101 has an electronic shutter function, required exposure timecan be assured under the control of a control signal from the controlunit 102. A flicker detection unit 103 detects a flicker component froma variation component of a signal level, which is included in an imagesignal output from the AD conversion unit 106 and caused by a flashinglight source.

The control unit 102 controls the entire system. The control unit 102outputs a control signal for controlling a accumulation time of theimage sensor 101 and outputs a control state of the image sensor 101 ascontrol information to a correction value prediction unit 104, whichwill be described below.

The correction value prediction unit 104 generates a correction valuefor correction so that a variation component of a signal level beinggenerated in an image signal does not appear, based on the flickercomponent detected in the flicker detection unit 103 and on the controlinformation output from the control unit 102 for controlling the imagesensor 101.

A flicker correction unit 105 executes arithmetic processing for animage signal based on the correction value generated in the correctionvalue prediction unit 104 and suppresses a flicker included in an imagesignal.

Next, referring to FIG. 2, a timing chart according to the firstexemplary embodiment of the present invention will be described.

A vertical synchronizing signal 201 changes according to a type of animaging apparatus. Reading drive of the image sensor 101 is representedby r1 to r7. In the present exemplary embodiment, a complementarymetal-oxide semiconductor (CMOS) image sensor is employed as the imagesensor 101, which employs an XY address type scanning method.

In the case of an image sensor of the XY address type scanning method,since an exposure for each pixel is sequentially executed along a linedirection, as shown in FIG. 2, all pixels of the image sensor 101 aresequentially read during a period of one vertical synchronizing signal.

In addition, s1 to s7 represent driving to reset an electric chargeaccumulated on an electric charge conversion unit of the image sensor101. In order to standardize an exposure time of each pixel, a resettingoperation is also sequentially executed in a line direction similar toreading drive.

The time interposed between s (illustrated by a dotted line) and r(illustrated by a solid line) shown in FIG. 2 indicates an accumulationtime of the image sensor 101. An image signal from A to C, in FIG. 2, isoutput as an image signal having a shorter accumulation time and animage signal from D to G is output as an image signal having a longeraccumulation time.

A timing signal 202 indicates the timing when the flicker detection unit103 executes a processing operation. The flicker detection unit 103executes a detection operation in a part of a period when an imagesignal is being read.

A timing signal 203 indicates the timing when the control unit 102outputs a control signal to the image sensor 101 for controllingthereof. In the case shown in FIG. 2, at timing 204, the control unit102 outputs a control signal b to control the accumulation time foracquiring the image signal D to the image sensor 101.

A timing signal 205 indicates the timing when the control unit 102outputs control information for the image sensor 101 to the correctionvalue prediction unit 104. The control information is output with adelay by one vertical synchronizing period compared with the timing whenthe control unit 102 outputs a control signal to the image sensor 101.In the case illustrated in FIG. 2, the control unit 102 outputs thecontrol signal b to the image sensor 101 at timing 204, while thecontrol unit 102 outputs the control signal b to the correction valueprediction unit 104 at timing 206. This allows the accumulation timewhen the image signal D is captured to be referred to when a correctionvalue to correct the image signal D is to be generated.

A timing 207 indicates the timing when the correction value predictionunit 104 predicts a correction value based on the control informationobtained from the control unit 102 and a flicker component obtained fromthe flicker detection unit 103. In FIG. 2, the image sensor 101 isdriven so as to abruptly change the accumulation time between imagesignals C and D. The correction value prediction unit 104 receivescontrol information which has been used to obtain the image signal,which is an object for flicker detection, and continuously comparescontrol information one vertical synchronizing period before and controlinformation to be obtained next. Then, when the control information ischanged, the correction value prediction unit 104 operates so as toexecute a prediction of a correction value.

Each of timings 208 to 214 indicates a timing when the correction valueprediction unit 104 sets a generated correction value to the flickercorrection unit 105. That is, the correction value prediction unit 104operates to generate a correction value based on a flicker componentobtained from the flicker detection unit 102 when the controlinformation is not changed. On the other hand, the correction valueprediction unit 104 operates to predict, based on a flicker componentobtained from the flicker detection unit 102 and control informationoutput from the control unit 102 to generate a correction value when thecontrol information is changed. The set correction value is updated bythe next vertical synchronizing signal and the correction operation isperformed on a next image signal that is read from the image sensor 101.In the case of FIG. 2, timing 210 indicates the timing when thecorrection value to be predicted and generated is set.

Next, a flicker component output from the flicker detection unit 103will be described together with an example of a detection operation.

FIGS. 3A and 3B schematically illustrate an image signal captured by theimage sensor 101. A state of an image signal 301 is obtained in a casewhen the accumulation time of the image sensor 101 is shorter. A stateof an image signal 303 is obtained in a case when the accumulation timeof the image sensor 101 is longer. Since a signal level is modulated bya flashing light source, a periodic level variation caused by a lightemission cycle of the flashing light source is generated in an imagesignal as indicated in signal level graphs 302 and 304. A periodicvariation of a signal level at this time dimension is converted into afrequency range to be detected, thereby allowing a flicker componentincluded in an image signal to be extracted.

There are several methods for converting an image signal obtained fromthe image sensor 101 into a frequency range. With respect to datasampled at a fixed time period as an image signal, discrete Fouriertransform is most commonly used to convert an image signal into afrequency range. A method will be described below in which an imagesignal is converted into a frequency range using discrete Fouriertransform to calculate a flicker component.

A flicker caused by a flashing light source appears to form light anddark stripes arranged in the vertical direction caused by the drivingsystem of the image sensor 101 (refer to FIGS. 3A and 3B). Therefore,sampling data generated in the vertical direction in an image signal isenough to detect a flicker component. For example, there is a method toaverage a signal for each horizontal line and reduce the signal intotwo-dimensional information of a time axis (time sampling) and a signallevel as illustrated by the signal level graphs 302 and 304.Alternatively, without limiting to these methods, when a flicker at thecenter of an image signal causes a user to feel uncomfortable, adetection area can be arranged at the center of an image to average asignal for each horizontal direction in the detection area. In any case,data reduced into two-dimensional information of a time axis (timesampling) and a signal level is generated.

Further, when the video signal type is National Television StandardsCode (NTCS) and the frequency of the commercial power supply is 50 Hz,since a flicker cycle which appears in an image signal is 100/60≈1.7cycles, it is sufficient to sample all or a portion of one cycle or morein the vertical direction of an image signal. Also, when the videosignal type is Phase Alternation by Line (PAL) and the frequency of thecommercial power supply is 60 Hz, since a flicker cycle is a 120/50=2.4cycles, similarly, it is sufficient to sample all or a portion of onecycle or more in the vertical direction of an image signal.

Next, an example will be described which converts the obtained samplingdata into a frequency range. When a population parameter of sampling isN and sampled data is Xn (n=0, 1, . . . N−1), data is discrete Fouriertransformed into a frequency range using a discrete Fourier coefficientrepresented by the following equation:

$\begin{matrix}{{{\overset{\sim}{x}}_{k} = {\frac{1}{N}{\sum\limits_{n = 0}^{N - 1}{x_{n} \cdot {\exp \left( {{- } \cdot \frac{2\; \pi \; {nk}}{N}} \right)}}}}}\left( {{k = 0},{{1\mspace{14mu} \ldots \mspace{14mu} N} - 1}} \right)} & (1)\end{matrix}$

When data is sampled N times each for time L (sec) and a samplinginterval is provided by a (sec), a Fourier coefficient after discreteFourier transform is determined within a range of a discrete frequencyfk represented by the following equation:

$\begin{matrix}{f_{k} = {\frac{k}{N} \times f_{s}\mspace{14mu} \left( {f_{s} = {\frac{1}{a} = \frac{N}{L}}} \right)}} & (2)\end{matrix}$

where fs is a sampling frequency in a time domain.

Since a data row sampled from an image signal is a real-time signal, ifequation (1) is expanded, equation (1) can be separated into a real part(Re) and an imaginary part (Im) and is represented by the followingequation:

$\begin{matrix}{{\overset{\sim}{x}}_{k} = {{{Re}\; \left( {\overset{\sim}{x}}_{k} \right)} + {j \times {{Im}\left( {\overset{\sim}{x}}_{k} \right)}}}} & (3) \\{{{Re}\; \left( {\overset{\sim}{x}}_{k} \right)} = {{\frac{1}{N}{\sum\limits_{n = 0}^{N - 1}{{x_{n} \cdot {\cos \left( \frac{2\; \pi \; {nk}}{N} \right)}}\mspace{14mu} {{Im}\left( {\overset{\sim}{x}}_{k} \right)}}}} = {{- \frac{1}{N}}{\sum\limits_{n = 0}^{N - 1}{x_{n} \cdot {\sin \left( \frac{2\; \pi \; {nk}}{N} \right)}}}}}} & (4)\end{matrix}$

An amplitude spectrum Ak in a certain frequency fk is determined by thefollowing equation according to equation (4).

|A _(k) |=√{square root over (Re({tilde over (x)} _(k))³ +Im({tilde over(x)} _(k))²)}  (5)

Further, a phase component Ωk in a certain frequency fk when convertedinto a frequency range is determined by the following equation accordingto equation (4).

$\begin{matrix}{\Omega_{k} = {\tan^{- 1}\left( {- \frac{{Im}\left( {\overset{\sim}{x}}_{k} \right)}{{Re}\left( {\overset{\sim}{x}}_{k} \right)}} \right)}} & (6)\end{matrix}$

From equation (5) and equation (6), it is understood that equation (4)contains a frequency component as a flicker component and a phasecomponent corresponding to the frequency component, and represents aflicker component that is output from the flicker detection unit 103 tothe correction value prediction unit 104.

Next, a method will be described which predicts a correction parameterbased on a flicker component (amplitude and phase at each frequency)obtained from the above-described flicker detection unit 103 and controlinformation on the image sensor 101 obtained from the control unit 102in the correction value prediction unit 104.

Specifically, a method will be described in which when the driving stateof the image sensor 101 is not changed, for example in FIG. 2, a flickercomponent in an image signal B is detected to generate a correctionvalue for correcting an image signal C. As described above, it is knownthat a flicker, which is generated due to a flashing light source,appears in an image signal with a double frequency of a power supplyfrequency. For example, when the imaging apparatus is an NTSC type andthe power supply frequency is 50 Hz, a flicker repeatedly appears forevery three fields in the same phase.

Hence, a phase difference θk between a phase of a flicker component of aflicker generated in the image signal C and a phase of a flickercomponent of a flicker generated in an image signal D is represented bythe following equation at k provided when a frequency fk calculated byequation (2) is fk=100 (Hz) and m provided when the phase difference θkrepresented by the following equation satisfies θk=2πfk×(1/60).

$\begin{matrix}{\vartheta_{k} = {{- \frac{2\; \pi \; m}{N}}k}} & (7)\end{matrix}$

An amplitude value of a flicker component depends on the accumulationtime of the image sensor 101. Since the accumulation time of the imagesignal B and that of the image. signal C are similar, the amplitudevalues can be regarded to have no variation between frames. Thus, aflicker component can be converted into a time domain by predictingphase components from among all flicker components detected in the imagesignal B by the amount indicated in equation (7) and by transforming thepredicted phase components using inverse discrete Fourier transform.

Inverse discrete Fourier transform is represented by the followingequation with a real part and an imaginary part of discrete Fouriertransform (equation (4)).

$\begin{matrix}{x_{n} = {\sum\limits_{k = 0}^{N - 1}\left\{ {{{{Re}\left( {\overset{\sim}{x}}_{k} \right)}{\cos \left( {\frac{2\; \pi \; {nk}}{N} + \vartheta_{k}} \right)}} - {{{Im}\left( {\overset{\sim}{x}}_{k} \right)}{\sin \left( {\frac{2\; \pi \; {nk}}{N} + \vartheta_{k}} \right)}}} \right\}}} & (8)\end{matrix}$

It is generally known that a flicker component is proportional to asignal strength of an object. In order to cancel a flicker to begenerated in an image signal, a flicker component of a time domaincalculated by the above equation, which is centered on 0 need to bechanged to be centered on 1. Then, by obtaining the inverse numberthereof, a correction value can be obtained. Then, multiplying an imagesignal by the generated correction value allows an image signal withoutany flicker to be generated.

On the other hand, referring again to FIG. 2, there is a timing when acorrection value is generated based on a flicker component detected inthe image signal C and a correction operation to the image signal D isexecuted. If a correction value is generated based on the flickercomponent detected from the image signal C having a shorter accumulationtime and the correction value is applied to the image signal D having alonger accumulation time, as apparent from the signal level graphs 302and 304 in FIGS. 3A and 3B, since amplitude values and phases of aflicker component are different from each other, a suitable correctionoperation cannot be executed.

Suitable correction can be executed on the image signal D by predictingan amplitude value and a phase based on the control information on theimage sensor 101 output from the control unit 102 when the correctionvalue is generated based on the detected flicker component.

A method for predicting a correction value will be described below. FIG.4 illustrates a relationship between an accumulation time of the imagesensor 101 and a modulation degree of a luminance signal caused by aflashing light source. The modulation degree is a value of a levelvariation (amplitude) when a flicker is generated by a flashing lightsource normalized by a signal level when a flicker is not generated.

A line 401 indicates a modulation degree of a signal level caused by aflicker for each accumulation time. As the accumulation time becomesshorter, the accumulation time of the light of a flashing light sourcebecomes shorter too. Consequently, the variation of a signal level foreach line becomes larger. Hence, the line 401 indicates thecharacteristics as shown in FIG. 4 due to the integration time and thecycle of a flashing light source.

The correction value prediction unit 104 has stored a table in advanceconcerning an accumulation time of the image sensor 101 and a modulationdegree of a luminance signal caused by a flashing light source. Thetable can be stored as discretized characteristics of the line 401 shownin FIG. 4, which is measured beforehand, or can be stored by learning inthe imaging apparatus.

When the accumulation time of the image sensor 101 is changed from t1 tot2, the correction value prediction unit 104 receives the accumulationtime t2 of the image sensor 101 from the control unit 102, as controlinformation of the image sensor 101, when an image signal to becorrected is accumulated. The correction value prediction unit 104acquires a past modulation degree a1 at the accumulation time t1 and amodulation degree a2 at the accumulation time t2 received currently byreferring to the table, and modulates a correction value determined byequation (8) by α2/α1 to generates a new correction value. Thereby,prediction of the amplitude of a correction value (correction parameter)can be obtained.

Referring to FIG. 5, a method for predicting a phase of a correctionvalue will be described.

Driving 503 and driving 505 indicate driving to reset an electric chargeaccumulated on an electric charge conversion portion of the image sensor101, and driving 504 and driving 506 indicate driving of reading. A timet1 between the driving 503 and the driving 504 and a time t2 between thedriving 505 and the driving 506 indicate accumulation times. In order tocalculate a phase difference in flicker components between theaccumulation time t1 and the accumulation time t2, a time difference Abetween centers of gravity 501 and 502 of the accumulation times foraccumulating a flashing light source is converted into phaseinformation. When the cycle of a vertical synchronizing signal is 1/60s, the time difference A is represented by the following equation:

$\begin{matrix}{A = {\frac{1}{60} + \left( \frac{{t\; 1} - {t\; 2}}{2} \right)}} & (9)\end{matrix}$

That is, the second term of equation (9) represents a phase differenceof a flicker component generated due to a change in accumulation time.Thus, when a correction value is calculated, in addition to the phasedifference θk of equation (7), a component of the second term ofequation (9) is added as a phase difference component Φk due to a changein accumulation time, thereby allowing prediction of a phase to beperformed.

The phase difference component Φk is represented by the followingequation including k when the frequency fk calculated using equation (2)is fk=100 (Hz) and 1 provided when the phase difference component Φksatisfies Φk=2πfk×((t1−t2)/2).

$\begin{matrix}{\Phi_{k} = {{- \frac{2\; \pi \; 1}{N}}k}} & (10)\end{matrix}$

As described above, a correction value can be calculated by using thefollowing equation according to equation (8) and equation (10) topredict a phase difference due to an accumulation time difference. Thus,a correction value can be obtained.

$\begin{matrix}{x_{n} = {\sum\limits_{k = 0}^{N - 1}\begin{Bmatrix}{{{Re}\left( {\overset{\sim}{x}}_{k} \right){\cos \left( {\frac{2\; \pi \; {nk}}{N} + \Phi_{k} + \vartheta_{k}} \right)}} -} \\{{Im}\left( {\overset{\sim}{x}}_{k} \right){\sin \left( {\frac{2\; \pi \; {nk}}{N} + \Phi_{k} + \vartheta_{k}} \right)}}\end{Bmatrix}}} & (11)\end{matrix}$

An exemplary embodiment has been described in which detection andcorrection are executed using discrete Fourier transform and inversediscrete Fourier transform in the whole range of a frequency range whichis determined by a sampling frequency fs. However, for example,referring to FIGS. 3A and 3B, it is understood that when theaccumulation time is longer, the variation is roughly similar to a sinewave with a frequency, at which a flashing light source is emittinglight. Accordingly, when a correction value is generated, the correctionvalue is generated using the frequency fk of the generated flicker andFourier series of a frequency of the vicinity thereof without executinginverse discrete Fourier transform over the entire frequency range,thereby allowing a sufficient correction value to be obtained.

On the other hand, if the accumulation time is becoming short, a flickergenerated on a image signal becomes roughly similar to a light emissionshape of a flashing light source. Thus, a flicker will be difficult tobe approximated by a simple sine wave. Higher harmonic components areincreased in the flicker. At this time, whether a flicker containshigher harmonic waves is determined according to the accumulation timewhich is control information of the image sensor 101 obtained from thecontrol unit 102.

The correction value prediction unit 104 stores an accumulation time ofthe image sensor 101 and a modulation degree of a luminance signalcaused by a flashing light source in a plurality of tables containing ahigher harmonic waves in advance. When inverse discrete Fouriertransform is performed to obtain a correction value, the correctionvalue prediction unit 104 performs inverse discrete Fourier transformadding higher harmonic components by referring to a modulation degreefrom the table so as to contain higher harmonic components correspondingto the accumulation time. Even if a flicker containing a higher harmonicwave component is generated, by performing the processing describedabove, appropriate flicker correction can be realized.

As described above, even if a flicker component is abruptly changed dueto a change in driving method of an image sensor, the flicker componentcan precisely be corrected and image quality can be enhanced.

Further, the present exemplary embodiment uses a complementarymetal-oxide semiconductor (CMOS) image sensor, which employs an XYaddress type scanning method. However, the present exemplary embodimentcan be applied to image sensors, which execute resetting and reading ofall pixels at the same time as a charge-coupled device (CCD). In thiscase, a flicker appears as a variation of a periodic signal levelbetween captured image signals. When the accumulation time of the imagesensor 101 is changed, the accumulation time and the modulation degreeof a luminance signal caused by a flashing light source are stored as atable, and a correction value is predicted and generated based oncontrol information output from the control unit 102, thereby allowingappropriate flicker correction to be realized.

Referring to FIG. 6, a timing chart according to a second exemplaryembodiment of the present invention will be described below. Descriptionis not repeated for those that are assigned the same reference numeralsas those of the first exemplary embodiment. The second exemplaryembodiment illustrates a method for detecting and correcting a flickerwhen an image signal, of which a flicker is corrected, has a delay bytwo vertical synchronizing signal periods relative to an image signal inwhich a flicker is detected.

Here, the case is described in which the image sensor 101 has changedthe operation state from an intermittent reading operation to a normalreading operation. A timing chart 601 indicates the timing when theflicker detection unit 103 is operated. In the timing chart 601, theflicker detection unit 103 is operated in a part of a period when animage signal is obtained. In the intermittent reading operation, theimage sensor 101 synchronizes with a vertical synchronizing period todetermine frames to be read or not to be read. The intermittent readingoperation is alternately repeated in a part illustrated in FIG. 6. Inthe normal reading operation an image signal is read from the imagesensor 101 for each vertical synchronizing period. That is, the imagingapparatus is driven so as to change a frame rate.

Further, the intermittent reading operation and the normal readingoperation will be described below when each accumulation time is notdifferent. A control signal 602 is output from the control unit 102 tothe image sensor 101. A control information signal 604 is output fromthe control unit 102 to the correction value prediction unit 104. Thetiming 604, when it is “high”, indicates the timing when the correctionvalue prediction unit 104 is operated, and timings 605 to 609, when theyare “high”, indicate the timing when a correction value generated by thecorrection value prediction unit 104 is set.

In this case, the timing of flicker detection and flicker correctionalways cause a delay of two vertical synchronizing periods betweendetection and correction, because flicker detection is executed from theimage signal A and correction is executed on the image signal B, orflicker detection is executed from the image signal B and correction isexecuted on the image signal C. Thus, when the driving state of theimage sensor 101 is changed, since an image signal to be detected isabsent, a correction value cannot be obtained. Thus, correction to theimage signal D cannot be performed. To solve this problem, the controlunit 102 and the correction value prediction unit 104 execute aprediction for the image signal D at a timing that will be describedbelow.

A control signal of the image sensor 101 is output from the control unit102 to the image sensor 101 at a timing shown in the control signal 602.In the control signal 602, a signal c indicates a driving signal fordriving the shutter without executing the reading drive, a signal dindicates a driving signal for executing reading drive without drivingthe shutter, and a signal e indicates a normal reading drive signal.

Here, when the driving state of the image sensor 101 is changed fromintermittent reading to normal reading by the control unit 102, attiming shown in the control signal 602, the driving signal e is outputto change the driving state of the image sensor 101. As shown in thecontrol information signal 603, the driving state of the image sensor101 is output, as control information, from the control unit 102 to thecorrection value prediction unit 104.

The control information includes information (in the present exemplaryembodiment, high and low) for detecting a frame rate change. Thecorrection value prediction unit 104 generates a correction value forthe image signal C based on a flicker component detected by the flickerdetection unit 102 from the image signal B. A phase of a flickergenerated in a detection frame and a phase of a flicker generated in acorrection frame have a fixed value at k provided when a frequency fkcalculated by equation (2) is fk=100 (Hz) and m provided when a phasedifference θk represented by equation (7) satisfies θk=2πfk×(1/30).

Further, assuming that the accumulation time of an image signal obtainedfrom the image sensor 101 is not changed, an amplitude spectrum of aflicker is changed little. Thus, according to equation (8), a correctionvalue for the image signal C is generated from a result detected fromthe image signal B and the correction value is set at a timing indicatedby a timing signal 606.

In this way, after a correction value for the image signal C isgenerated, by receiving information indicating that the driving state ofthe image sensor 101 is changed (change from intermittent reading tonormal reading) from the control information shown in the controlinformation signal 603 and acquiring information beforehand that acorrection value for the image signal D is missing due to the change ofthe driving state, a correction value for the image signal D isgenerated based on a flicker component detected from the image signal Band the control information signal 603 for controlling the image sensor101 output from the control unit 102. A flicker component detected fromthe image signal B is continuously stored in a storage unit. When thecontrol information output from the control unit 102 is changed, thecontrol information is read from the storage unit. After a correctionvalue for the image signal C is generated, a correction value for theimage signal D is predicted and generated at a timing indicated by atiming signal 604. The generated correction value is set at a timingindicated by a timing signal 607 to correct the image signal D.

Next, a method for predicting a phase of a correction value to beexecuted in the correction value prediction unit 104 in the presentexemplary embodiment will be described.

When an image signal to be corrected is delayed by two verticalsynchronizing periods relative to an image signal from which a flickeris detected, as described above, a phase difference between a flicker inthe detection frame and a flicker to be corrected in the correctionframe has a fixed value at k provided when a frequency fk calculated byequation (2) is fk=100 (Hz) and m provided when a phase difference θkrepresented by equation (7) satisfies θk=2πfk×(1/30).

As in the present exemplary embodiment, when a correction value forcorrecting a flicker of the image signal D is generated from a resultdetected from the image signal B, a phase difference can be factored inthe phase difference component Φk by one vertical synchronizing period.Then, the correction value has a fixed value at k provided when afrequency fk calculated using equation (2) is fk=100 (Hz) and 1 providedwhen the phase difference component Φk represented by equation (10)satisfies Φk=2πfk×(1/60). Therefore, a correction value is generatedfrom equation (11), thereby allowing appropriate flicker correction tobe realized.

Referring to the drawings, a third exemplary embodiment of the presentinvention will be described below. Since a block diagram of a flickerdetection and correction apparatus is similar to that of the firstexemplary embodiment, description is not repeated.

Next, referring to FIG. 7, a timing chart according to the thirdexemplary embodiment of the present invention will be described.

The present exemplary embodiment is an example of control for changing adriving method of an imaging method in order to capture a still imagehaving higher definition than a moving image, for example, duringcapturing a moving image. For example, there is a method in which when amoving image is captured, in order to read the moving image at a highspeed, an image is read from an image sensor by adding and averaging orthinning it. On the other hand, when a still image is captured, in orderto read a high-definition image, an image is read using a readout timelonger than that the time to be used for a moving image by reducing thenumber of adding and averaging or thinning operations than the numberfor a moving image, or by omitting addition.

Signal 701 is a vertical synchronizing signal which is determinedaccording to an image capturing method of the imaging apparatus. In thecase of an image sensor using an XY address type scanning method, sincean exposure for each pixel is sequentially executed in the linedirection, as illustrated in FIG. 7, all pixels of an image sensor aresequentially read within one vertical synchronizing signal period.

Reading drive of the image sensor 101 is represented by r1 to r6.Further, driving to reset an electric charge accumulated on an electriccharge conversion portion of the image sensor 101 is represented by s1to s6. In order to standardize an exposure time of each pixel, aresetting operation is also sequentially executed in the line directionsimilar to the reading operation.

As illustrated in FIG. 7, the reading time of r4 is longer than that ofeach of r1, r2, r3, r5, and r6. As described above, this represents thatthe reading time is changed due to a change in driving method of theimage sensor, for example, by switching between image capturing of amoving image and a still image. An image signal read at timing r4corresponds to an image signal D in FIG. 7, and image signals read attimings r1, r2, r3, r5, and r6 correspond to image signals A, B, C, E,and F, respectively.

FIGS. 8A and 8B schematically illustrate an image signal captured by theimage sensor 101. A state 801 illustrates a state of an image signalwhen the reading time for reading from the image sensor 101 is shorterduring moving image capturing. A state 803 illustrates a state of animage signal when the reading time for reading from the image sensor 101is longer during still image capturing. In fact, since the number ofpixels to be read for each timing is different, the size of an image isdifferent. However, in order to compare the cycle of a flicker at thesame image angle for each case, the image signals are illustrated in thesame size.

Since a signal level is modulated by a flashing light source, as shownin states 802 and 804, a periodic level variation depending on a lightemission cycle of the flashing light source occurs in an image signal.If the light emission cycle of the flashing light source is constant,since the cycle of a level variation that appears in a screen depends onthe reading time, the cycle of the level variation can be predicted byusing reading time information as control information.

Referring back to FIG. 7, a control method will be described.

Timing 702 indicates the timing when a processing operation of theflicker detection unit 103 is executed. In timing 702, the flickerdetection unit 103 executes a detection operation in a part of a periodof time when an image signal is read. Timing 703 indicates the timingwhen the control unit 102 outputs a control signal for controlling theimage sensor 101 thereto and is used to execute a driving method whichaffects the reading time, such as an adding, an averaging and a thinningmethod in an image sensor.

In the case of FIG. 7, for example, at timing 704, a control signal ffor capturing the image signal C as a moving image is transmitted to theimage sensor 101, and the image sensor 101 executes resetting andreading at timings s3 and r3, respectively. At timing 705, a controlsignal g for capturing the image signal D as a still image istransmitted to the image sensor 101, and the image sensor 101 executesresetting and reading at timings s4 and r4, respectively. At timing 706,a control signal f, which represents that driving is to be continued, istransmitted to the image sensor 101, and the image sensor 101 continuesresetting and reading at timings s4 and r4, respectively. At timing 707,a control signal f for capturing an image signal E as a still image istransmitted to the image sensor 101, and the image sensor 101 executesresetting and reading at timings s5 and r5, respectively.

In the present exemplary embodiment, for image signals B, C, and F whichare obtained when a moving image is captured, a correction value foreach of the signals B, C, and F is generated from a flicker componentobtained from image signals A, B, and E, respectively, which arecaptured just before image signals B, C, and F, respectively, and can beset at timings 713, 714, and 717.

For the image signal D obtained when a still image is captured, sincethe state of the flicker is different from that of the image signal Cobtained just before the image signal D, a correction value is predictedfrom the flicker component obtained from the image signal C. Further,for an image signal E obtained when a moving image is captured, sincethe state of a flicker is different from that of the image signal Dobtained just before the image signal E, a correction value is predictedbased on a flicker component obtained from the image signal D.

Information 708 is control information that the control unit 102transmits to the correction value prediction unit 104 and represents aratio of reading time. At timing 709, a ratio of reading times r3 and r4is transmitted as control information i, and at timing 710, a ratio ofreading times r4 and r5 is transmitted as control information j. Thecorrection value prediction unit 104 predicts a correction value attimings 711 and 712 based on a flicker component and controlinformation, and sets a correction value at timings 715 and 716. Inother fields, a correction value is calculated based on a flickercomponent to set a correction value.

Next, a method executed by the correction value prediction unit 104 willbe described which predicts a correction value based on the flickercomponent obtained from the above-described flicker detection unit 103and control information obtained from the control unit 102. A methodthat determines a correction value from a flicker component usinginverse discrete Fourier transform to be represented by equation (8) issimilar to the first exemplary embodiment.

If a reading time of a field 1 for reading from the image sensor 101 isprovided by t1 and a reading time of another field 2 is provided by t2,a ratio of the cycle of a level variation, in a screen, of field 1 tofield 2 is represented by a ratio of reading time T=t1/t2.

Thus, prediction of a correction value to a cyclic component can beperformed using the above-described T as control information, morespecifically, as represented by equation (12), by performing dividingprocessing for each term with T.

$\begin{matrix}{x_{n} = {\sum\limits_{k = 0}^{N - 1}\begin{Bmatrix}{{{Re}\left( {\overset{\sim}{x}}_{k} \right){\cos \left( {{\frac{1}{T} \times \frac{2\; \pi \; {nk}}{N}} + \vartheta_{k}} \right)}} -} \\{{Im}\left( {\overset{\sim}{x}}_{k} \right){\sin \left( {{\frac{1}{T} \times \frac{2\; \pi \; {nk}}{N}} + \vartheta_{k}} \right)}}\end{Bmatrix}}} & (12)\end{matrix}$

As described above, even when a flicker component is abruptly changeddue to a change in driving method of the imaging sensor 101, a flickercomponent can precisely be corrected and image quality can be enhanced.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application No.2007-340390 filed Dec. 28, 2007, which is hereby incorporated byreference herein in its entirety.

1. An image processing apparatus comprising: a change unit configured tochange a driving state of an image sensor for capturing an image signalin response to an operation for changing the driving state of the imagesensor; and a flicker correction unit configured to correct an imagesignal based on a flicker component detected from other image signals ofa frame captured a predetermined number of frames before the frame ofthe image signal to be corrected, wherein the flicker correction unit isconfigured to change the predetermined number when the driving state ofthe image sensor is changed by the change unit.
 2. The image processingapparatus according to claim 1, wherein the change in the driving stateof the image sensor is a change in frame rate, and wherein the flickercorrection unit is configured to increase the predetermined number asthe frame rate is increased.
 3. The image processing apparatus accordingto claim 1, wherein when the driving state of the image sensor changesfrom the driving state of the moving image capturing to the drivingstate of the still image capturing, and further changes to the drivingstate of the moving image, the flicker correction unit corrects an imagesignal of a newly captured moving image based on a flicker componentdetected from the previously captured frame of the image signal in thedriving state of the moving image capturing.
 4. An image processingapparatus comprising: a control unit configured to control a drivingstate of an image sensor for capturing an image signal; and a flickercorrection unit configured to detect a flicker component from the imagesignal captured by the image sensor and to correct the image signalbased on the detected flicker component, wherein the flicker correctionunit is configured to execute different corrections to the image signalsdepending on whether a driving state of the image sensor is maintainedin a first state or the driving state of the image sensor is transferredfrom the first state to a second state.
 5. The image processingapparatus according to claim 4, wherein the flicker correction unit isconfigured to predict a higher harmonic component included in the imagesignal to correct a flicker component of an image signal.
 6. An imageprocessing apparatus comprising: a change unit configured to change adriving state of an image sensor for capturing an image signal inresponse to an operation for changing the driving state of the imagesensor; and a flicker correction unit configured to correct an imagesignal based on a flicker component detected from other image signals ofa frame captured before the frame of the image signal to be corrected,wherein the flicker correction unit generates a flicker correction valueby predicting a flicker component of a image signal to be correctedbased on a flicker component of the image signal captured before theframe of the image signal to be corrected when the driving state of theimage sensor is changed.
 7. The image processing apparatus according toclaim 6, wherein the flicker correction unit generates the flickercorrection value for the image to be corrected based on the accumulationtime of the image to be corrected and of the image captured before theimage to be corrected.
 8. The image processing apparatus according toclaim 6, wherein the flicker correction unit generates the flickercorrection value by predicting at least one of an amplitude value and aphase of the flicker component.
 9. A method comprising: changing adriving sate of an image sensor for capturing an image signal inresponse to an operation for changing the driving state of the imagesensor; correcting the image signal based on a flicker componentdetected from image signals of a frame captured a predetermined numberof frames before the frame of the image signal to be corrected; andchanging the predetermined number when the driving state of the imagesensor is changed.
 10. A method comprising: detecting a flickercomponent from an image signal captured by an image sensor; correctingthe image signal based on the detected flicker component; and executinga different correction to the image signal between a period when thedriving state of the image sensor is maintained in a first state and atransfer period when the driving state of the image sensor istransferring to a second state different from the first state and is notyet maintained in the second state.
 11. A method comprising: changing adriving sate of an image sensor for capturing an image signal inresponse to an operation for changing the driving state of the imagesensor; predicting a flicker component of a image signal to becorrected, when the driving state of the image sensor is changed, basedon a flicker component of a image signal captured therebefore;correcting the image signal to be corrected according to the predictedflicker component.